Pixel arrangements for image sensors with phase detection pixels

ABSTRACT

An image sensor may include a pixel array having image pixels for capturing image data and phase detection pixels for gathering phase information during automatic focusing operations. Phase detection pixels may form phase detection pixel pairs having first and second pixels with different angular responses. The first and second pixels may have color filters of the same color or may have color filters of different colors. The phase detection pixel pairs may be isolated from other phase detection pixel pairs in the array or may be arranged consecutively in a line. The phase detection pixels may, for example, be provided with color filters to match the color filter pattern of the pixel array. Processing circuitry may adjust red and green pixel signals from a phase detection pixel pair having a red and green color filter and may subsequently determine a phase difference using the adjusted pixel signals.

BACKGROUND

This relates generally to imaging systems and, more particularly, toimaging systems with phase detection capabilities.

Modern electronic devices such as cellular telephones, cameras, andcomputers often use digital image sensors. Imager sensors (sometimesreferred to as imagers) may be formed from a two-dimensional array ofimage sensing pixels. Each pixel receives incident photons (light) andconverts the photons into electrical signals. Image sensors aresometimes designed to provide images to electronic devices using a JointPhotographic Experts Group (JPEG) format.

Some applications such as automatic focusing and three-dimensional (3D)imaging may require electronic devices to provide stereo and/or depthsensing capabilities. For example, to bring an object of interest intofocus for an image capture, an electronic device may need to identifythe distances between the electronic device and object of interest. Toidentify distances, conventional electronic devices use complexarrangements. Some arrangements require the use of multiple imagesensors and camera lenses that capture images from various viewpoints.Other arrangements require the addition of lenticular arrays that focusincident light on sub-regions of a two-dimensional pixel array. Due tothe addition of components such as additional image sensors or complexlens arrays, these arrangements lead to reduced spatial resolution,increased cost, and increased complexity.

Some electronic devices include both image pixels and phase detectionpixels in an a single image sensor. With this type of arrangement, acamera can use the on-chip phase detection pixels to focus an imagewithout requiring a separate phase detection sensor. In a typicalarrangement, phase detection pixels all have the same color and arearranged consecutively in a line in the pixel array. When phasedetection pixels are arranged in this way, optical crosstalk becomesproblematic. For example, optical crosstalk between a green image pixeland a green phase detection pixel may be more difficult to correct thatoptical crosstalk between a green image pixel and a red image pixel.

It would therefore be desirable to be able to provide improved phasedetection pixel arrangements and phase detection signal processingmethods for image sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an illustrative electronic device withan image sensor that may include phase detection pixels in accordancewith an embodiment of the present invention.

FIG. 2A is a cross-sectional view of illustrative phase detection pixelshaving photosensitive regions with different and asymmetric angularresponses in accordance with an embodiment of the present invention.

FIGS. 2B and 2C are cross-sectional views of the phase detection pixelsof FIG. 2A in accordance with an embodiment of the present invention.

FIG. 3 is a diagram of illustrative signal outputs of phase detectionpixels for incident light striking the phase detection pixels at varyingangles of incidence in accordance with an embodiment of the presentinvention.

FIG. 4A is a top view of an illustrative phase detection pixel pairarranged horizontally in accordance with an embodiment of the presentinvention.

FIG. 4B is a top view of an illustrative phase detection pixel pairarranged vertically in accordance with an embodiment of the presentinvention.

FIG. 4C is a top view of an illustrative phase detection pixel pairarranged vertically and configured to detect phase differences along thehorizontal direction in accordance with an embodiment of the presentinvention.

FIG. 5 is a top view of a conventional pixel array having phasedetection pixels arranged consecutively in a line and all having thesame color.

FIG. 6 is a top view of an illustrative pixel array having isolatedpairs of phase detection pixels with color filters that match the colorfilter pattern of the pixel array in accordance with an embodiment ofthe present invention.

FIG. 7 is a top view of an illustrative pixel array having pairs ofphase detection pixels arranged consecutively in a line with colorfilters that match the color filter pattern of the pixel array inaccordance with an embodiment of the present invention.

FIG. 8 is a top view of an illustrative pixel array having isolatedpairs of stacked phase detection pixels with color filters that matchthe color filter pattern of the pixel array in accordance with anembodiment of the present invention.

FIG. 9 is a top view of an illustrative pixel array having pairs ofstacked phase detection pixels arranged consecutively in a line withcolor filters that match the color filter pattern of the pixel array inaccordance with an embodiment of the present invention.

FIG. 10 is a top view of an illustrative pixel array having isolatedpairs of phase detection pixels arranged non-consecutively in a linewith color filters that match the color filter pattern of the pixelarray in accordance with an embodiment of the present invention.

FIG. 11 is a top view of an illustrative pixel array having isolatedpairs of diagonally-oriented phase detection pixels with color filtersthat match the color filter pattern of the pixel array in accordancewith an embodiment of the present invention.

FIG. 12 is a top view of an illustrative pixel array having alternatingpairs of red phase detection pixels and green phase detection pixelsarranged in a line in accordance with an embodiment of the presentinvention.

FIG. 13 is a top view of an illustrative pixel array having pairs ofgreen phase detection pixels distributed along two or more pixel rows inaccordance with an embodiment of the present invention.

FIG. 14 is a flowchart of illustrative steps involved in operating animage sensor having image pixels and phase detection pixels inaccordance with an embodiment of the present invention.

FIG. 15 is a flowchart of illustrative steps involved in processingphase detection pixel signals during automatic focusing operations inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention relate to image sensors withautomatic focusing and depth sensing capabilities. An electronic devicewith a camera module is shown in FIG. 1. Electronic device 10 may be adigital camera, a computer, a cellular telephone, a medical device, orother electronic device. Camera module 12 (sometimes referred to as animaging device) may include one or more image sensors 14 and one or morelenses 28. During operation, lenses 28 (sometimes referred to as optics28) focus light onto image sensor 14. Image sensor 14 includesphotosensitive elements (e.g., pixels) that convert the light intodigital data. Image sensors may have any number of pixels (e.g.,hundreds, thousands, millions, or more). A typical image sensor may, forexample, have millions of pixels (e.g., megapixels). As examples, imagesensor 14 may include bias circuitry (e.g., source follower loadcircuits), sample and hold circuitry, correlated double sampling (CDS)circuitry, amplifier circuitry, analog-to-digital (ADC) convertercircuitry, data output circuitry, memory (e.g., buffer circuitry),address circuitry, etc.

Still and video image data from image sensor 14 may be provided to imageprocessing and data formatting circuitry 16. Image processing and dataformatting circuitry 16 may be used to perform image processingfunctions such as automatic focusing functions, depth sensing, dataformatting, adjusting white balance and exposure, implementing videoimage stabilization, face detection, etc. For example, during automaticfocusing operations, image processing and data formatting circuitry 16may process data gathered by phase detection pixels in image sensor 14to determine the magnitude and direction of lens movement (e.g.,movement of lens 28) needed to bring an object of interest into focus.

Image processing and data formatting circuitry 16 may also be used tocompress raw camera image files if desired (e.g., to Joint PhotographicExperts Group or JPEG format). In a typical arrangement, which issometimes referred to as a system on chip (SOC) arrangement, camerasensor 14 and image processing and data formatting circuitry 16 areimplemented on a common integrated circuit. The use of a singleintegrated circuit to implement camera sensor 14 and image processingand data formatting circuitry 16 can help to reduce costs. This is,however, merely illustrative. If desired, camera sensor 14 and imageprocessing and data formatting circuitry 16 may be implemented usingseparate integrated circuits.

Camera module 12 may convey acquired image data to host subsystems 20over path 18 (e.g., image processing and data formatting circuitry 16may convey image data to subsystems 20). Electronic device 10 typicallyprovides a user with numerous high-level functions. In a computer oradvanced cellular telephone, for example, a user may be provided withthe ability to run user applications. To implement these functions, hostsubsystem 20 of electronic device 10 may include storage and processingcircuitry 24 and input-output devices 22 such as keypads, input-outputports, joysticks, and displays. Storage and processing circuitry 24 mayinclude volatile and nonvolatile memory (e.g., random-access memory,flash memory, hard drives, solid state drives, etc.). Storage andprocessing circuitry 24 may also include microprocessors,microcontrollers, digital signal processors, application specificintegrated circuits, or other processing circuits.

It may be desirable to provide image sensors with depth sensingcapabilities (e.g., to use in automatic focusing applications, 3Dimaging applications such as machine vision applications, etc.). Toprovide depth sensing capabilities, image sensor 14 may include phasedetection pixel groups such as pixel pair 100 shown in FIG. 2A.

FIG. 2A is an illustrative cross-sectional view of pixel pair 100. Pixelpair 100 may include first and second pixels such Pixel 1 and Pixel 2.Pixel 1 and Pixel 2 may include photosensitive regions 110 formed in asubstrate such as silicon substrate 108. For example, Pixel 1 mayinclude an associated photosensitive region such as photodiode PD1, andPixel 2 may include an associated photosensitive region such asphotodiode PD2. A microlens may be formed over photodiodes PD1 and PD2and may be used to direct incident light towards photodiodes PD1 andPD2. The arrangement of FIG. 2A in which microlens 102 covers two pixelregions may sometimes be referred to as a 2×1 or 1×2 arrangement becausethere are two phase detection pixels arranged consecutively in a line.

Color filters such as color filter elements 104 may be interposedbetween microlens 102 and substrate 108. Color filter elements 104 mayfilter incident light by only allowing predetermined wavelengths to passthrough color filter elements 104 (e.g., color filter 104 may only betransparent to the certain ranges of wavelengths). Photodiodes PD1 andPD2 may serve to absorb incident light focused by microlens 102 andproduce pixel signals that correspond to the amount of incident lightabsorbed.

Photodiodes PD1 and PD2 may each cover approximately half of thesubstrate area under microlens 102 (as an example). By only coveringhalf of the substrate area, each photosensitive region may be providedwith an asymmetric angular response (e.g., photodiode PD1 may producedifferent image signals based on the angle at which incident lightreaches pixel pair 100). The angle at which incident light reaches pixelpair 100 relative to a normal axis 116 (i.e., the angle at whichincident light strikes microlens 102 relative to the optical axis 116 oflens 102) may be herein referred to as the incident angle or angle ofincidence.

An image sensor can be formed using front side illumination imagerarrangements (e.g., when circuitry such as metal interconnect circuitryis interposed between the microlens and photosensitive regions) or backside illumination imager arrangements (e.g., when photosensitive regionsare interposed between the microlens and the metal interconnectcircuitry). The example of FIGS. 2A, 2B, and 2C in which pixels 1 and 2are backside illuminated image sensor pixels is merely illustrative. Ifdesired, pixels 1 and 2 may be front side illuminated image sensorpixels. Arrangements in which pixels are backside illuminated imagesensor pixels are sometimes described herein as an example.

In the example of FIG. 2B, incident light 113 may originate from theleft of normal axis 116 and may reach pixel pair 100 with an angle 114relative to normal axis 116. Angle 114 may be a negative angle ofincident light. Incident light 113 that reaches microlens 102 at anegative angle such as angle 114 may be focused towards photodiode PD2.In this scenario, photodiode PD2 may produce relatively high imagesignals, whereas photodiode PD1 may produce relatively low image signals(e.g., because incident light 113 is not focused towards photodiodePD1).

In the example of FIG. 2C, incident light 113 may originate from theright of normal axis 116 and reach pixel pair 100 with an angle 118relative to normal axis 116. Angle 118 may be a positive angle ofincident light. Incident light that reaches microlens 102 at a positiveangle such as angle 118 may be focused towards photodiode PD1 (e.g., thelight is not focused towards photodiode PD2). In this scenario,photodiode PD2 may produce an image signal output that is relativelylow, whereas photodiode PD1 may produce an image signal output that isrelatively high.

The positions of photodiodes PD1 and PD2 may sometimes be referred to asasymmetric positions because the center of each photosensitive area 110is offset from (i.e., not aligned with) optical axis 116 of microlens102. Due to the asymmetric formation of individual photodiodes PD1 andPD2 in substrate 108, each photosensitive area 110 may have anasymmetric angular response (e.g., the signal output produced by eachphotodiode 110 in response to incident light with a given intensity mayvary based on an angle of incidence). In the diagram of FIG. 3, anexample of the pixel signal outputs of photodiodes PD1 and PD2 of pixelpair 100 in response to varying angles of incident light is shown.

Line 160 may represent the output image signal for photodiode PD2whereas line 162 may represent the output image signal for photodiodePD1. For negative angles of incidence, the output image signal forphotodiode PD2 may increase (e.g., because incident light is focusedonto photodiode PD2) and the output image signal for photodiode PD1 maydecrease (e.g., because incident light is focused away from photodiodePD1). For positive angles of incidence, the output image signal forphotodiode PD2 may be relatively small and the output image signal forphotodiode PD1 may be relatively large.

The size and location of photodiodes PD1 and PD2 of pixel pair 100 ofFIGS. 2A, 2B, and 2C are merely illustrative. If desired, the edges ofphotodiodes PD1 and PD2 may be located at the center of pixel pair 100or may be shifted slightly away from the center of pixel pair 100 in anydirection. If desired, photodiodes 110 may be decreased in size to coverless than half of the pixel area.

Output signals from pixel pairs such as pixel pair 100 may be used toadjust the optics (e.g., one or more lenses such as lenses 28 of FIG. 1)in image sensor 14 during automatic focusing operations. The directionand magnitude of lens movement needed to bring an object of interestinto focus may be determined based on the output signals from pixelpairs 100.

For example, by creating pairs of pixels that are sensitive to lightfrom one side of the lens or the other, a phase difference can bedetermined. This phase difference may be used to determine both how farand in which direction the image sensor optics should be adjusted tobring the object of interest into focus.

When an object is in focus, light from both sides of the image sensoroptics converges to create a focused image. When an object is out offocus, the images projected by two sides of the optics do not overlapbecause they are out of phase with one another. By creating pairs ofpixels where each pixel is sensitive to light from one side of the lensor the other, a phase difference can be determined. This phasedifference can be used to determine the direction and magnitude ofoptics movement needed to bring the images into phase and thereby focusthe object of interest. Pixel groups that are used to determine phasedifference information such as pixel pair 100 are sometimes referred toherein as phase detection pixels or depth-sensing pixels.

A phase difference signal may be calculated by comparing the outputpixel signal of PD1 with that of PD2. For example, a phase differencesignal for pixel pair 100 may be determined by subtracting the pixelsignal output of PD1 from the pixel signal output of PD2 (e.g., bysubtracting line 162 from line 160). For an object at a distance that isless than the focused object distance, the phase difference signal maybe negative. For an object at a distance that is greater than thefocused object distance, the phase difference signal may be positive.This information may be used to automatically adjust the image sensoroptics to bring the object of interest into focus (e.g., by bringing thepixel signals into phase with one another).

Pixel pairs 100 may arranged in various ways. For example, as shown inFIG. 4A, Pixel 1 (referred to herein as P1) and Pixel 2 (referred toherein as P2) of pixel pair 100 may be oriented horizontally, parallelto the x-axis of FIG. 4A (e.g., may be located in the same row of apixel array). In the example of FIG. 4B, P1 and P2 are orientedvertically, parallel to the y-axis of FIG. 4B (e.g., in the same columnof a pixel array). In the example of FIG. 4C, P1 and P2 are arrangedvertically and are configured to detect phase differences in thehorizontal direction (e.g., using an opaque light shielding layer suchas metal mask 30). Various arrangements for phase detection pixels aredescribed in detail in U.S. patent application Ser. No. 14/267,695,filed May 1, 2014, which is hereby incorporated by reference herein inits entirety.

A typical arrangement for phase detection pixels is shown in FIG. 5. Theconventional pixel array 500 of FIG. 5 includes an array of image pixels502. Phase detection pixel pairs 504 in pixel array 500 are arrangedconsecutively in a line. Pixel array 500 includes a color filter array.Pixels marked with an R include a red color filter, pixels marked with aG include a green color filter, and pixels marked with a B include ablue color filter. The pattern of color filters in image pixels 502 is aBayer mosaic pattern which includes a repeating unit cell of two-by-twoimage pixels 502 having two green image pixels arranged on one diagonaland one red and one blue image pixel arranged on the other diagonal. Asshown in FIG. 5, phase detection pixel pairs 504 are all formed withgreen color filter elements, which disrupts the Bayer mosaic pattern ofpixel array 500.

When the color filter pattern is disrupted in this way, replacing phasedetection pixel signals with interpolated image pixel values can bechallenging. Optical crosstalk between image pixels 502 and 504 can alsobecome problematic, as algorithms that correct for optical crosstalk inpixel arrays with a particular type of color filter pattern are lesseffective in correcting optical crosstalk when the color filter patternis disrupted.

To overcome these challenges, phase detection pixels may be arrangedsuch that disruption of the color filter pattern in the pixel array isminimized. Illustrative phase detection pixel arrangements that minimizethe amount by which the color filter pattern is altered are shown inFIGS. 6-11. In the examples of FIGS. 6-13, pixels marked with an Rinclude a red color filter, pixels marked with a G include a green colorfilter, and pixels marked with a B include a blue color filter. Dashedlines such as dashed line 102M may indicate regions that are covered bya single microlens such as microlens 102 of FIG. 2A.

The use of red, green, and blue color filters in FIGS. 6-13 is merelyillustrative. If desired, the color filter patterns may includebroadband filters. For example, each two-by-two unit of pixels mayinclude one pixel having a broadband filter. In general, any suitablecolor filter pattern and any suitable type of color filter may be usedin image sensor 14. The examples of FIGS. 6-13 are merely illustrative.

As shown in FIG. 6, pixel array 32 may include image pixels 34 and phasedetection pixels 36. Pixel array 32 may include an array of color filterelements such as red color filter elements (e.g., color filter materialthat passes red light while reflecting and/or absorbing other colors oflight), blue color filter elements (e.g., color filter material thatpasses blue light while reflecting and/or absorbing other colors oflight), green color filter elements (e.g., color filter material thatpasses green light while reflecting and/or absorbing other colors oflight), yellow color filter elements (e.g., yellow color filter materialthat passes red and green light), clear color filter elements (e.g.,transparent material that passes red, blue, and green light), broadbandfilter elements (e.g., filter material that passes two or more colors oflight selected from the group that includes red light, blue light, andgreen light), and/or color filter elements of other colors (e.g., cyan,magenta, etc.).

In the example of FIG. 6, phase detection pixel pairs 100 are formed ina scattered arrangement in which each phase detection pixel pair isisolated from other phase detection pixel pairs. For example, each phasedetection pixel pair 100 may be substantially or completely surroundedby image pixels 34. By surrounding phase detection pixel pairs 100 withimage pixels 34, replacing the phase detection pixel values withinterpolated image pixel values during the image reconstruction processmay be facilitated by the greater number of neighboring image pixels 34around each phase detection pixel pair 100. Scattered phase detectionpixel pairs 100 may be oriented horizontally in a row of pixel array 32(as shown in the example of FIG. 6) or may be oriented vertically in acolumn of pixel array 32. Pixel array 32 may include one, two, three,ten, less than ten, or more than ten phase detection pixel pairs 100,and the pixel pairs 100 may be positioned at any suitable location inarray 32.

Pixel pairs 100 may include color filter elements to match the colorfilter pattern of image pixels 34. In the example of FIG. 6, imagepixels 34 include color filter elements that form a Bayer pattern, andphase detection pixels 36 include color filter elements that do notdisrupt the Bayer pattern. For example, P1 of phase detection pixel pair100 may include a green color filter element and P2 of phase detectionpixel pair 100 may include a red color filter element. This is, however,merely illustrative. If desired, P1 may include a red color filterelement and P2 may include a green color filter element or P1 mayinclude a blue color filter element and P2 may include a green colorfilter element (as examples). The color filter elements of P1 and P2 arechosen to match the color filter pattern of the entire pixel array 32.

The example of FIG. 6 in which the color filter pattern of pixel array32 is a Bayer color filter pattern is merely illustrative. If desired,other suitable patterns of color filters may be used (e.g., pseudo-Bayercolor filter patterns in which one or both of the green color filters ineach two-by-two unit is replaced with a different type of filter such asa broadband filter, other suitable color filter patterns, etc.). Thecolor filter elements formed in phase detection pixel pairs 36 may beany suitable color so long as the color filter pattern of the pixelarray 32 is unchanged. Arrangements in which the color filter pattern ofpixel array 32 is a Bayer color filter pattern are sometimes describedherein as an example.

In the example of FIG. 7, phase detection pixel pairs 100 are arrangedconsecutively in a line (e.g., a line segment that includes two or moreconsecutive pixel pairs 100). Phase detection pixel pairs 100 may bearranged horizontally in a row of pixel array 32 (as shown in theexample of FIG. 7) or may be arranged vertically in a column of pixelarray 32. As with the example of FIG. 6, image pixels 34 of FIG. 7include color filter elements that form a Bayer pattern, and phasedetection pixels 36 include color filter elements that leave the Bayerpattern intact. For example, P1 of each pixel pair 100 may include agreen color filter element and P2 of each pixel pair 100 may include ared color filter element. In general, the color filter elements formedin phase detection pixel pairs 36 may be any suitable color so long asthe color filter pattern of pixel array 32 is unchanged.

In the example of FIGS. 8, P1 and P2 of each phase detection pixel pair100 are located diagonally from each other in separate rows and separatecolumns of pixel array 32. This is sometimes referred to as a stackedphase detection pixel arrangement because the microlens 102 that coversP1 of pixel pair 100 is formed above or below the microlens 102 thatcovers P2 of pixel pair 100.

As with the example of FIG. 6, phase detection pixel pairs 100 of FIG. 8are isolated from other phase detection pixel pairs in array 32. Forexample, each phase detection pixel pair 100 may be substantially orcompletely surrounded by image pixels 34. By surrounding phase detectionpixel pairs 100 with image pixels 34, replacing the phase detectionpixel values with interpolated image pixel values during the imagereconstruction process may be facilitated by the greater number ofneighboring image pixels 34 around each phase detection pixel pair 100.

Image pixels 34 of FIG. 8 include color filter elements that form aBayer pattern, and phase detection pixels 36 include color filterelements that leave the Bayer pattern intact. However, the stackedarrangement of FIG. 8 allows P1 and P2 of each phase detection pixelpair to be the same color. For example, in cases where the color filterpattern is a Bayer color filter pattern, P1 of each pixel pair 100 mayinclude a green color filter element and P2 (located diagonally from P1)of each pixel pair 100 may also include a green color filter element. Ingeneral, the color filter elements formed in phase detection pixel pairs36 may be any suitable color so long as the color filter pattern ofpixel array 32 is unchanged.

In the example of FIG. 9, phase detection pixel pairs 100 having thestacked arrangement of the type shown in FIG. 8 are arrangedconsecutively in a line (e.g., a line segment that includes two or moreadjacent pixel pairs 100). Phase detection pixel pairs 100 may bearranged in two consecutive rows of pixel array 32 (as shown in theexample of FIG. 9) or may be arranged in two consecutive columns ofpixel array 32. As with the example of FIG. 8, image pixels 34 of FIG. 9include color filter elements that form a Bayer pattern, and phasedetection pixels 36 include color filter elements that leave the Bayerpattern intact. For example, P1 of each pixel pair 100 may include agreen color filter element and P2 (located diagonally from P1) of eachpixel pair 100 may also include a green color filter element. Ingeneral, the color filter elements formed in phase detection pixel pairs36 may be any suitable color so long as the color filter pattern ofpixel array 32 is unchanged.

In the example of FIG. 10, pixel pairs 100 are arrangednon-consecutively in a line (e.g., are located in a common row) and areisolated from each other by image pixels 34. This type of arrangement issimilar to that of FIG. 6 in that each phase detection pixel pair 100may be substantially or completely surrounded by image pixels 34. Bysurrounding phase detection pixel pairs 100 with image pixels 34,replacing the phase detection pixel values with interpolated image pixelvalues during the image reconstruction process may be facilitated by thegreater number of neighboring image pixels 34 around each phasedetection pixel pair 100.

FIG. 11 is a variation on the example of FIG. 8 in which P1 and P2 ofeach phase detection pixel pair 100 are located diagonally from eachother in separate rows and separate columns of pixel array 32. In theexample of FIG. 11, however, the microlens 102M formed over P1 and themicrolens formed over P2 are shifted with respect to one another (e.g.,are located diagonally from one another rather than being stacked as inthe example of FIG. 8).

Phase detection pixel pairs 100 of the type shown in FIG. 11 may beisolated from other phase detection pixel pairs 100. For example, eachphase detection pixel pair 100 may be substantially or completelysurrounded by image pixels 34. By surrounding phase detection pixelpairs 100 with image pixels 34, replacing the phase detection pixelvalues with interpolated image pixel values during the imagereconstruction process may be facilitated by the greater number ofneighboring image pixels 34 around each phase detection pixel pair 100.This is, however, merely illustrative. If desired, phase detection pixelpairs 100 of the type shown in FIG. 11 may be arranged in a line (e.g.,there may be two or more pixel pairs 100 located adjacent to one anotheralong a line).

Image pixels 34 of FIG. 11 include color filter elements that form aBayer pattern, and phase detection pixels 36 include color filterelements that leave the Bayer pattern intact. However, the arrangementof FIG. 11 allows P1 and P2 of each phase detection pixel pair to be thesame color. For example, in cases where the color filter pattern is aBayer color filter pattern, P1 of each pixel pair 100 may include agreen color filter element and P2 (located diagonally from P1) of eachpixel pair 100 may also include a green color filter element. Ingeneral, the color filter elements formed in phase detection pixel pairs36 may be any suitable color so long as the color filter pattern ofpixel array 32 is unchanged.

FIGS. 12 and 13 are examples in which the color filter pattern of pixelarray 32 is slightly altered by the color filters of phase detectionpixels 36. In the example of FIGS. 12, P1 and P2 of a phase detectionpixel pair 100 have the same color (e.g., red or green), and the pixelpairs 100 are arranged consecutively in a line (e.g., a line segmentthat includes two or more adjacent pixel pairs 100). However, unlike theconventional arrangement of FIG. 5, the color of pixel pairs 100 in theline changes from one pair to the next pair. For example, a pair ofgreen phase detection pixels 36 may be interposed between first andsecond pairs of red phase detection pixels 36. Alternating pairs ofgreen phase detection pixels with pairs of red phase detection pixels inthis way may facilitate the image reconstruction process when phasedetection pixel values are replaced with interpolated image pixelvalues. For example, a red image pixel value for a pair of green phasedetection pixels may be determined based at least partly on the redpixel values from neighboring red phase detection pixels.

In the example of FIG. 13, phase detection pixels 36 are all the samecolor but are separated into different rows of pixel array 32. Forexample, a first row of pixels may include one or more phase detectionpixel pairs 100 arranged in a non-consecutive line, whereas a second rowof pixels may include one or more phase detection pixel pairs 100arranged in a non-consecutive line. P1 and P2 of each pixel pair 100 maybe the same color such as green (as an example). As shown in FIG. 13,pixel pairs 100 in one row may be staggered with respect to the pixelpairs 100 in another row.

The example of FIG. 13 in which pixel pairs 100 are formed in two rowsof pixel array 32 is merely illustrative. If desired, pixel pairs 100may be formed in more than two rows (e.g., four rows, six rows, etc.).Pixel pairs 100 in each row may be staggered with respect to pixel pairs100 in the adjacent rows.

If desired, the phase detection pixel values from P1 and P2 may be usedduring the reconstruction process during which the phase detection pixelvalues from P1 and P2 are replaced with interpolated image pixel values.Consider P1′ and P2′ of FIG. 13, for example. To replace the phasedetection pixel value P1 from pixel P1′ with a green pixel value G andto replace the phase detection pixel value P2 from pixel P2′ with a redpixel value R, a green estimate G′ may first be determined using thefollowing formula:

G′=k(x, y)*(P1+P2)  (1)

where k(x,y) is a spatially-dependent coefficient or gain function(e.g., determined during a calibration procedure) that minimizes apredetermined function. For example, the following function may beminimized:

$\begin{matrix}{\sum\limits_{x,y}\left( {\frac{G_{1} + G_{2}}{2} - G^{\prime}} \right)^{2}} & (2)\end{matrix}$

where G₁ and G₂ are average green pixel values inside a fixed kernelM×M, and where M is a constant (e.g., M=9). The above function shown in(2) is an illustrative example of a particular function that may beminimized to obtain coefficient k(x,y). If desired, other functions maybe used.

Average green pixel values may be determined based on weighted sums ofnearby green pixels. After determining a green estimate G′, localgradients may be calculated as follows:

G _(A) −G′|,|G _(B) −G′|,|G _(R) −G′|,|G _(L) −G′|  (3)

where G_(A), G_(B), G_(R), and G_(L) are average green pixel values at anearby locations above pixels P1′ and P2′, below pixels P1′ and P2′, tothe right of pixels P1′ and P2′, and to the left of pixels P1′ and P2′,respectively.

The missing pixel values R and G are then interpolated based onsimilarity to neighboring pixels. For example, the missing G value couldbe set equal to the pixel value of a neighboring green pixel having thesmallest associated local gradient. This value may then be used inreconstructing the missing red value R, in a manner similar to thatemployed in chroma-luma demosaic algorithms. If desired, the outputpixel value G may be determined by clipping the green estimate G′:

G=clip(G′,min(G _(A) ,G _(B) ,G _(R) ,G _(L)),max(G _(A) ,G _(B) ,G _(R),G _(L)))  (4)

FIG. 14 is a flowchart of illustrative steps involved in operating animage sensor such as image sensor 14 of FIG. 1 having image pixels 34and phase detection pixels 36.

At step 200, image pixel array 32 may gather data from the light that isfocused onto pixel array 32. For example, image pixels 34 may gatherimage data and phase detection pixels 36 may gather phase informationfrom incoming light. Phase detection pixels 36 may, for example, producepixel signals of the type shown in FIG. 3. Image data and phaseinformation from image pixels and phase detection pixels may be providedto image processing circuitry 16.

At step 204, processing circuitry 16 may process the gathered phaseinformation and image data to determine whether one or more objects inthe image are in focus. In one suitable arrangement, a user ofelectronic device 10 may select an object of interest in the imagedscene (sometimes referred to as an interest point) and processingcircuitry 16 may determine whether the object of interest is in focus.In another suitable arrangement, processing circuitry 16 mayautonomously identify objects of interest (interest points) in theimaged scene such as a face, a person, a moving object, or any otherdesired object using the image data captured by image pixels 34 and thephase information captured by phase detection pixels 36. For example,processing circuitry 16 may implement a face detection algorithm toautomatically detect faces in an imaged scene and to identify thosefaces as interest points. Processing circuitry 16 may determine whetherthe identified object of interest is in focus (e.g., using the phasedata gathered by phase detection pixels 36).

For example, processing circuitry 16 may determine whether objects inthe imaged scene are in focus by comparing angular pixel outputs from P1and P2 of a phase detection pixel pair such as outputs of the type shownin FIG. 3. The algorithm used by processing circuitry 16 to determine aphase difference associated with phase detection pixel signals dependson how phase detection pixels are arranged in the array. Forconventional arrangements of the type shown in FIG. 5 in which phasedetection pixels are all the same color and are arranged consecutivelyin a line, a cross-correlation algorithm is used to determine a phasedifference. A similar cross-correlation algorithm may be used to processphase detection pixel signals from phase detection pixels having anarrangement of the type shown in FIG. 9.

In arrangements of the type shown in FIG. 10 in which phase detectionpixel pairs 100 are arranged non-consecutively in a row and include P1and P2 of different colors, a modified cross-correlation algorithm maybe used. For example, the processing circuitry 16 may perform a firstcross-correlation algorithm using pixel signals from red phase detectionpixels in the row and a second cross-correlation algorithm using pixelsignals from green phase detection pixels in the row. The red datacross-correlation results may be merged with the green datacross-correlation results to determine a phase difference.

In arrangements of the type shown in FIG. 7 in which phase detectionpixel pairs 100 are arranged consecutively in a line (e.g., a linesegment that includes two or more adjacent pixel pairs 100), again amodified cross-correlation algorithm may be used. However, rather thanperforming a separate cross-correlation operation for each color channel(as described above), the cross-correlation operation may be performedbetween colors. This may include, for example, normalizing the line ofpixel data from pixel pairs 100 and subsequently adjusting thenormalized pixel signals to reduce any bias between pixel signals fromred phase detection pixels and pixel signals from green phase detectionpixels. For example, pixel data from red phase detection pixels may beadjusted using predetermined correction factors or using pixel signalsfrom nearby phase detection pixels in essentially uniform areas so thatthe red pixel data becomes comparable to the green pixel data. Afterminimizing the difference in signal levels from red and green phasedetection pixels, a cross-correlation algorithm may be performed todetermine a phase difference.

In arrangements of the type shown in FIGS. 6, 8, 11, 12, and 13, adifferent algorithm may be used whereby a brightness gradient (sometimesreferred to as edge polarity) associated with an edge in an image isinferred based on pixel signals from nearby pixels (e.g., nearby imagepixels 34 and/or nearby phase detection pixels 36). This type ofalgorithm is sometimes referred to as edge polarity inference and isdescribed in greater detail below in connection with FIG. 15.

If processing circuitry 16 determines that the image is not in focus(e.g., that the objects of interest in the imaged scene are not infocus), processing may proceed to step 206 to adjust the focus of lens28 (FIG. 1). This may include, for example, determining a direction (andmagnitude, if desired) of lens movement needed to bring the image intofocus. Processing may then loop back to step 200 so that processingcircuitry 16 can determine whether or not the scene or objects ofinterest are in focus when captured using the new focal length.

If processing circuitry 16 determines in step 204 that the image is infocus, processing may proceed to step 208.

At step 208, processing circuitry 16 may generate a focused image usingthe image data captured by imaging pixels 34. This may include, forexample, replacing the phase detection pixel values with interpolatedimage pixel values using pixel values from nearby image pixels 34 (e.g.,image pixels 34 in the vicinity of phase detection pixels 36). Ifdesired, phase detection pixel values may be replaced with pixel valuesthat are interpolated using other phase detection pixels (e.g., forarrangements of the type shown in FIG. 12). The image reconstructionprocess may include, for example, using formulas 1, 2, 3, and 4 asdescribed above in connection with FIG. 13 to interpolate the missingpixel values for phase detection pixels 36.

Processing circuitry 16 may perform additional image processingoperations on the focused image in step 208 (e.g., white balanceoperations, noise correction operations, gamma correction operations,etc.). If desired, processing circuitry 16 may store the focused imagein memory and/or convey the focused image to additional processingcircuitry or display equipment. In this way, phase detection pixels 36may generate data that is used to automatically adjust the position oflens 28 to bring one or more objects of interest into focus. If desired,phase detection pixels 36 may be used to generate range information suchas one or more depth maps or other three dimensional information aboutan imaged scene.

FIG. 15 is a flowchart of illustrative steps involved in processingphase detection pixel signals during automatic focusing operations whenphase detection pixels are isolated (e.g., not arranged consecutively toform a substantial line segment). Pixels P1 and P2 of a phase detectionpixel pair 100 may be different colors or may be the same color.

At step 300, phase detection pixels 36 (e.g., P1 and P2 of FIG. 6, 8,11, 12, or 13) may gather phase information from a scene. Phasedetection pixel signals from P1 and P2 may be provided to processingcircuitry 16.

At step 302, processing circuitry 16 may optionally adjust the pixelvalues from P1 and P2 using a spatially-dependent calibration factor(e.g., k(x,y) of equation 1) The spatially-dependent calibration factormay be used to map phase detection pixel values from phase detectionpixels P1 and P2 to adjusted pixel values that correspond to pixelvalues that would be produced by P1 and P2 if P1 and P2 were normalimage pixels (as opposed to pixels with asymmetric angular responses).

At step 304, processing circuitry 16 may optionally determine and applya color adjustment to compensate for differences in color between P1 andP2. For example, processing circuitry 16 may adjust pixel values from P1and P2 based on white balance gains and/or based on other information.

At step 306, processing circuitry 16 may determine a phase differenceassociated with the pixel values from phase detection pixels P1 and P2.For example, processing circuitry may determine a phase difference P_(D)by subtracting P1 from P2, where P1 and P2 are pixel signals from phasedetection pixels P1 and P2.

Step 306 may optionally include a compensation operation whereby thephase difference P_(D)=P1−P2 is corrected to compensate for inherentsignal differences between P1 and P2. For example, pixels P1 and P2 mayproduce different signals when P1 and P2 of a phase detection pixel pair100 are different colors. P1 and P2 may also produce different signalseven when P1 and P2 are the same color as a result of manufacturingvariations, color filter material mismatch, and/or quantum efficiencydifferences (e.g., resulting from optical crosstalk from image pixels 34to phase detection pixels 36).

To perform the optional compensation operation, processing circuitry 16may look for phase detection pixels in nearby “flat” regions. Forexample, a pixel pair 100 in a flat region (e.g., a region withoutedges) may include phase detection pixels P1 _(f) and P2 _(f). Since theregion is flat, processing circuitry 16 assumes that P1 _(f)=P2 _(f)regardless of focus. Processing circuitry 16 may then determine a biasvalue bias(x,y) using P1 _(f) and P2 _(f) in the flat region:

$\begin{matrix}{{{{bias}\left( {x,y} \right)} = \frac{{P\; 1_{f}} - {P\; 2_{f}}}{{P\; 1_{f}} + {P\; 2_{f}}}},{{{when}\mspace{14mu} {\left( \overset{\rightharpoonup}{\nabla} \right)_{G}}} < {th}_{bias}}} & (5)\end{matrix}$

where ∇ _(G) is a gradient and th_(bias) is a fixed value characteristicof the imaging system and P1 _(f) and P2 _(f) are pixel signals from P1_(f) and P2 _(f), respectively. The corrected phase difference P_(D)′can then be calculated in the following way:

P _(D)′=(P1−P2)−bias(x,y)*(P1+P2)  (6)

At step 308, processing circuitry 16 may determine a brightness gradientassociated with an edge at phase detection pixels 36 (e.g., an edge thatis perpendicular to an axis passing through P1 and P2). The brightnessgradient of the edge may be determined using pixel values from nearbyimage pixels 34. For example, a gradient may be determined as ∇_(G)=(∇_(x),∇_(y)) if and only if the following is true

∇_(x) >th ₁,∇_(y) <th ₂*∇_(x)

P1−P2>∇_(x)*(th ₃ +th ₄)  (7)

where th₁, th₂, th₃, and th₄ are fixed values characteristic of theimaging system, ∇_(x) is a brightness gradient along a horizontaldirection in the pixel array, ∇_(y) is a brightness gradient along avertical direction in the pixel array, and P1 and P2 are pixel signalsfrom phase detection pixels P1 and P2.

At step 310, processing circuitry 16 may use the edge brightnessgradient determined in step 308 to determine the direction and, ifdesired, the magnitude of lens movement needed to bring the object ofinterest into focus. For example, the following autofocus (AF) logic maybe used to determine the direction of movement needed (if any):

sign(∇_(x))*(P1−P2)<0 move closer

sign(∇_(x))*(P1−P2)>0 move farther

sign(∇_(x))*(P1−P2)=0 in focus  (8)

Lens displacement may be approximated using the following formula:

$\begin{matrix}{{lens\_ displacement} \cong {f\left( \frac{{P\; 1} - {P\; 2}}{\nabla_{x}} \right)}} & (9)\end{matrix}$

where f is a function of the lens system.

Various embodiments have been described illustrating image sensor pixelarrays having image pixels for capturing image data and phase detectionpixels for gathering phase information during automatic focusingoperations. Phase detection pixels may form phase detection pixel pairshaving first and second pixels with different angular responses (e.g.,inverted, approximately inverted, or complementary angular responses).The first and second pixels may have color filters of the same color ormay have color filters of different colors. The phase detection pixelpairs may be isolated from each other (e.g., may be partially orcompletely surrounded by image pixels) or may be arranged consecutivelyin a line.

The phase detection pixel pairs may be arranged in an image pixel arrayhaving a plurality of image pixels. The image pixel array may include acolor filter array having a particular color filter pattern. The phasedetection pixels may be provided with color filters to match the colorfilter pattern of the image pixel array. In other embodiments, the phasedetection pixels may be provided with color filters that slightly alterthe color filter pattern but that are patterned to facilitate the imagereconstruction process in which the phase detection pixel signals arereplaced with pixel signals.

Processing circuitry may be used to determine a phase differenceassociated with pixel signals from a phase detection pixel pair. Theprocessing circuitry may adjust the pixel signals to reduce any biasbetween the phase detection pixel signals. For example, the processingcircuitry may adjust the phase detection pixel signals usingspatially-dependent predetermined correction factors. In anothersuitable embodiment, the processing circuitry may adjust the phasedetection pixel signals using a bias value that is determined duringimage capture operations. The bias value may be determined using nearbyphase detection pixels. For example, the processing circuitry may usephase detection pixels in a flat region of an image to determine a biasvalue which is then used to reduce any bias between the phase detectionpixel signals that are being used for autofocus operations.

Processing circuitry may replace phase detection pixel values withinterpolated image pixel values during an image reconstruction process.The interpolated image pixel values may be determined based on abrightness gradient. The brightness gradient may be determined usingnearby image pixels. For example, to replace a green phase detectionpixel value with a green image pixel value, a local brightness gradientmay be determined using nearby green image pixels.

The foregoing is merely illustrative of the principles of this inventionand various modifications can be made by those skilled in the artwithout departing from the scope and spirit of the invention.

What is claimed is:
 1. An image sensor having a pixel array, wherein thepixel array comprises: a plurality of image pixels that gather imagedata; a pair of phase detection pixels that gather phase information,wherein the pair of phase detection pixels includes first and secondphase detection pixels with different angular responses; and a colorfilter array, wherein the color filter array comprises a plurality ofcolor filter elements formed over the plurality of image pixels, whereinthe plurality of color filter elements are arranged according to a colorpattern, and wherein the first and second phase detection pixels includerespective first and second color filter elements having respectivefirst and second colors that match the color pattern.
 2. The imagesensor defined in claim 1 wherein the color pattern is a Bayer colorpattern.
 3. The image sensor defined in claim 1 wherein the plurality ofcolor filter elements includes broadband filter elements.
 4. The imagesensor defined in claim 1 wherein the first and second color filterelements transmit light of different colors.
 5. The image sensor definedin claim 4 further comprising a plurality of additional pairs of phasedetection pixels, wherein the pair of phase detection pixels and theplurality of additional pairs of phase detection pixels are located in apixel row, and wherein each of the additional pairs of phase detectionpixels includes color filter elements that transmit light of differentcolors.
 6. The image sensor defined in claim 1 wherein the first andsecond phase detection pixels include respective first and secondphotodiodes that are covered by a single microlens.
 7. The image sensordefined in claim 1 wherein the first and second phase detection pixelsinclude respective first and second photodiodes that are each partiallycovered by an opaque shielding layer.
 8. The image sensor defined inclaim 1 wherein the pixel array comprises pixel rows and pixel columnsand wherein the first and second phase detection pixels are locatedadjacent to one another in one of the pixel rows.
 9. The image sensordefined in claim 1 wherein the pixel array comprises pixel rows andpixel columns and wherein the first and second phase detection pixelsare located adjacent to one another in one of the pixel columns.
 10. Theimage sensor defined in claim 1 wherein the first phase detection pixelincludes a first photodiode covered by a first microlens, wherein thesecond phase detection pixel includes a second photodiode covered by asecond microlens, and wherein the first and second phase detectionpixels are positioned diagonally from one another.
 11. The image sensordefined in claim 10 wherein the first microlens spans first and secondcolumns of the pixel array and wherein the second microlens is adjacentto the first microlens and also spans the first and second columns ofthe pixel array.
 12. The image sensor defined in claim 10 wherein thefirst microlens spans first and second columns in the pixel array andwherein the second microlens spans third and fourth columns of the pixelarray.
 13. An image sensor having a pixel array, wherein the pixel arraycomprises: a plurality of image pixels that gather image data; pairs ofphase detection pixels that gather phase information, wherein each pairof phase detection pixels includes first and second phase detectionpixels with different angular responses and wherein each pair of phasedetection pixels is completely surrounded by the image pixels and isisolated from the other pairs of phase detection pixels in the pixelarray; and a color filter array, wherein the color filter arraycomprises a plurality of color filter elements formed over the pluralityof image pixels, wherein the plurality of color filter elements arearranged according to a color pattern, and wherein each pair of phasedetection pixels include respective first and second color filterelements having respective first and second colors that match the colorpattern.
 14. The image sensor defined in claim 13 wherein the pairs ofphase detection pixels are arranged in a row and wherein the pairs ofphase detection pixels in the row are separated from each other by imagepixels in the row.
 15. The image sensor defined in claim 13 wherein thepairs of phase detection pixels include first pairs of phase detectionpixels in a first row of the pixel array and second pairs of phasedetection pixels in a second row of the pixel array and wherein thefirst pairs of phase detection pixels are staggered with respect to thesecond pairs of phase detection pixels.
 16. An image sensor having apixel array, wherein the pixel array comprises: a plurality of imagepixels that gather image data; and a pair of phase detection pixels thatgather phase information, wherein the pair of phase detection pixelsincludes first and second phase detection pixels with different angularresponses, wherein the first and second phase detection pixels includerespective first and second color filter elements, wherein the first andsecond color filter elements have respective first and second colors,and wherein the first color is different from the second color.
 17. Theimage sensor defined in claim 16 further comprising: a plurality ofcolor filter elements formed over the plurality of image pixels, whereinthe plurality of color filter elements are arranged according to a colorpattern, and wherein the first and second colors match the colorpattern.
 18. The image sensor defined in claim 17 wherein the colorpattern is a Bayer color pattern.
 19. The image sensor defined in claim16 wherein the first and second phase detection pixels includerespective first and second photodiodes that are covered by a singlemicrolens.
 20. The image sensor defined in claim 16 wherein the firstand second phase detection pixels include respective first and secondphotodiodes that are each partially covered by an opaque shieldinglayer.